FEAL: Fine-Grained Evaluation of Active Learning in Collaborative Learning Spaces

Numerous studies have shown the effectiveness of collaborative active-learning pedagogies compared to traditional lectures across STEM fields. However, incorporating active learning in large classes presents unique challenges in stimulating student engagement and developing quality activities. A growing trend at universities is to create collaborative learning spaces (CLSs) that are purposefully designed and equipped to facilitate active learning. While some research has identified some effective learning strategies for CLS environments based on learning psychology, the success of individual activities is neither defined nor measured. This gap in knowledge is often met with a trial and error approach over numerous semesters. Activity adjustments are made solely based on the instructors' partial perceptions, whereas activity effectiveness is neither directly evaluated nor correlated to student performance. We present a novel measurement instrument called Fine-grained Evaluation of Active Learning (FEAL) for large CLS-based classes. FEAL can be quickly administered by preceptors to record key measures of activity success such as student engagement, activity difficulty, activity time, and associated lecture time. Other relevant information such as the concepts covered by the activity and the activity type are also recorded to be later cross-analyzed. FEAL can be applied to code exam questions and to assess student performance for the same concepts. We applied FEAL to a large freshman-level computer programming course with an enrollment of 200 students over the course of one semester. We present an overview of FEAL, its administration process within the CLS, and a detailed account of our evaluation methodology. We also highlight key lessons learned on the engagement and success achieved by individual activities, and outline planned improvements to in-class activities based on the obtained results. Assessment of Collaborative Learning Numerous studies have demonstrated the effectiveness of collaborative active-learning pedagogies compared to traditional lectures across STEM fields [1][2][3][4] and computer science education in particular [5][6][7]. Active-learning techniques include think-pair-share exercises [8][9], peer instruction [10], group problem solving, activities in CLS environments and extensive discussions, among others. Incorporating active learning in large classes (greater than 100 students) presents unique challenges in classroom management, teaching assistant and preceptor training, and student engagement. Additionally, most active-learning techniques involve extensive student interaction. However, the predominant design of classroom spaces has focused on lecture-style pedagogies, which further impedes the effective adoption of active learning. Toward alleviating some of these challenges, a growing trend at universities is to create collaborative learning spaces (CLSs) that are purposefully designed and equipped to facilitate active learning [11]. While some research has identified learning activities that are effective in CLS environments [12] building upon learning psychology and general activity structure, the success of individual activities is neither defined nor measured. As such, instructors are left with little guidance on the quality and effectiveness of the activities that they design. This gap in knowledge is often met with a trial and error approach over numerous semesters and adjustments are made based on instructors' partial perceptions. Most importantly, activity effectiveness is not directly evaluated or correlated to student performance. Several classroom measurement instruments have been proposed for evaluating teaching practices. However, none can directly assess individual activities. The Teaching Dimensions Observation Protocol (TDOP) [13] documents classroom behaviors by periodically marking which of 46 behaviors (or codes) were observed in the classroom. Code categories are divided into teaching methods, teacher-student dialog, instructional technology, pedagogical strategies, student engagement, and other groupings. Observations are recorded periodically (i.e., every two minutes), and provide a high-level overview of teaching, whereas a more fine-grained observation is needed to assess activities. Additionally, the TDOP requires three days of training to ensure inter-rater reliability. The Classroom Observation Protocol for Undergraduate STEM (COPUS) [14] was designed to code how both instructors and students spend class time and requires minimal training (less than 2 hours). COPUS enables instructors to understand what percentage of time they lecture, pose questions, write on a whiteboard, answer student questions with entire class involved, hold one-on-one discussions, etc. Moreover, COPUS records the percentage of time students are listening, working individually, engaged in group discussions, etc. COPUS can be very effective for assessing how different teaching practices are being used within the classroom. It provides a measure of the extent that active-learning pedagogies are applied, but the quality of the activities performed by students is not assessed. Importantly, most observation tools are meant to be used by non-experts in the subject matter, requiring very little training. In contrast, providing observations on the quality of individual activities requires expertise to gauge the difficulty or appropriateness of a given activity. To address these limitations, we present an activity quality measurement instrument called Fine-grained Evaluation of Active Learning (FEAL). FEAL can be quickly administered by preceptors to record key measures of activity success such as student engagement, student success, activity difficulty, activity time, and associated lecture time. The instrument is designed to require minimal training and minimal effort within the classroom for recording observations. Quick administration of the instrument is critical because the preceptors recording with FEAL are primarily tasked with engaging the students with the given activities. A key difference of FEAL with other tools is that the preceptors must have expertise on the subject matter, as the intent is to evaluate activity quality. Moreover, relevant information such as the concepts covered by the activity are recorded and analyzed. The instrument is further applied to code exam questions accordingly and is used to correlate student performance for the same concepts. We applied FEAL to a large freshman-level computer programming course with an enrollment of 200 students over the course of one semester. The course was taught within a CLS by a team of two instructors and eight preceptors. The application of FEAL on this course demonstrates a positive correlation between the student engagement and exam performance, and the student engagement is higher when the introduced concepts are more difficult. Moreover, the time spent on activities does not necessarily yield better student performance in the exam. FEAL also enables an instructor to quickly identify outliers from the expectations (e.g., recursion activities) that may need to be redesigned, such as by adding more activity difficulty to enhance student engagement and to match difficulty level with the questions in the exam. Observing and Assessing In-class Collaborative Learning Activities Using FEAL Class Information and Operation We applied FEAL at a freshman-level programming CS1 course at a four-year university, using the ​C programming language. The typical course enrollment for one section is about 200 students. The course primarily serves the College of Engineering, but students from other colleges are frequently enrolled. Although the course is intended for freshmen, it is equally attended by sophomores, and juniors. Some senior and graduate students (primarily outside the College of Engineering) are also enrolled. The class is taught in a CLS with a maximum capacity of 260 students. Students are organized in round tables of up to six persons. Each table is equipped with 1-2 whiteboards and a table number. An A-type whiteboard is also available per three tables. The space is further equipped with over 20 screens placed around the room so that projected material is visible from every table and angle. The CLS layout, as it is seen from the instructor’s station point of view, is shown in Figure 1. Fig. 1. ​ Instructor view of the Collaborative Learning Space. The class consists of three main components: (a) reading assignments using the zyBooks online interactive book platform [15], (b) 75 minutes in-class sessions held twice a week, and (c) a 3-hour lab held weekly. Students are requested to complete a set of participation and challenge questions before every in-class session. These are automatically graded through the zyBooks platform. The in-class time is structured as a sequence of active-learning tasks, and lecturing/ demonstration periods. The administration of the activities is assisted by preceptors (teaching assistants and undergraduate learning assistants that have previously taken the course). A typical distribution of the instructors' and students’ activities during a 75 minutes class session, as it is recorded with the COPUS tool, is shown in Figure 2. The COPUS tool verifies that the majority of the in-class time is spent on group activities, polling questions, demonstrations, group guiding, and student questions, whereas little time is devoted to traditional lecturing. However, the COPUS graphs cannot be used to infer the activity quality. Fig. 2. ​ Overview of instructor's and student's time in a single class session using COPUS. (a) Instructor's time (b) Student’s time